Selection and presentation of context-relevant supplemental content and advertising

ABSTRACT

Media content is paired with context-relevant supplemental content, and the media and supplemental content are provided to a user. A media stream containing the media content may be received from a source system, and context information about the media content is determined from information about the media stream. The supplemental content may be selected based on the determined context information. This may enable a business model in which third parties can register advertising or other supplemental content and specify the criteria that cause it to be combined with the media content.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional applicationSer. No. 11/683,862 filed Mar. 8, 2007, now U.S. Pat. No. 7,769,756 B2issued Aug. 3, 2010, which claims the benefit of U.S. ProvisionalApplication No. 60/780,709, filed on Mar. 8, 2006, which is herebyincorporated by reference in its entirety; application Ser. No.11/683,862 is also a continuation-in-part of U.S. application Ser. No.11/147,664, filed on Jun. 7, 2005, which claims the benefit of U.S.Provisional Application No. 60/577,833, filed Jun. 7, 2004, both ofwhich are hereby incorporated by reference in their entirety.

BACKGROUND

1. Field of the Invention

This invention relates generally to media applications, and inparticular to presenting context-relevant supplemental content in amedia player.

2. Background of the Invention

Television and other media content are increasingly finding a home onnew devices. For example, television programming is no longer foundsolely on the television. Through computer networks, as well as otherdelivery mechanisms, media content can be consumed on electronicdevices, such as personal computers and cellular phones.

Media content is often provided with supplemental content to the users(e.g., viewers, consumers) of the media content. This supplementalcontent can generate income for the content provider (e.g., advertisingrevenue for providing advertisements in television programs) and/orprovide additional information to the users (e.g. display news scrollbaron top of a television program). However, the supplemental contentprovided generally is not relevant to the context of the particular userconsuming the particular media content (e.g., the content of the mediacontent, the user's preference, etc.), and therefore not of muchinterest to the user.

Accordingly, there is a need for a system and method to selectcontext-relevant supplemental content for media content and present thesupplemental content to users of the media content.

SUMMARY

Embodiments of the invention provide a system and method to selectcontext-relevant supplemental content for media content and present thecontext-relevant supplemental content and the media content to users. Inone embodiment, context information of the media content is determined;supplemental content is selected based on the context information andpresent to the users.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram of a process for presenting context-relevantsupplemental content to a user, according to one embodiment of theinvention.

FIG. 2 is a system that includes a source system, a context detectionand supplemental content selection system and a destination system,according to one embodiment of the invention.

FIG. 3 is a flowchart of a method for selecting and presentingcontext-relevant supplemental content for a media stream, according toone embodiment of the invention.

One skilled in the art will readily recognize from the followingdiscussion that alternative embodiments of the structures and methodsillustrated herein may be employed without departing from the principlesof the invention described herein.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Overview

Embodiments of the invention include systems and methods to pair mediacontent with context-relevant (or context-specific) supplemental contentand present the media content and the supplemental content to a user. Inthe following description, media content may include, withoutlimitation, one or a combination of audio (including music, radiobroadcasts, recordings, advertisements, etc.), video (including movies,video clips, television broadcasts, advertisements, etc.), data (such asstreaming weather data), software (including video games, multimediaprograms, graphics software), text and pictures. Media content may be inanalog or digital form, and may be delivered as one or more individualfiles (e.g., media clip) or one or more media streams. Media streams canbe encoded using any encoding technology and may be encrypted to protecttheir content using any suitable encryption technique.

Similar to media content, supplemental content may include one or acombination of the types of media content described above. One exampleof the supplemental content including a combination of different typesof media content is a multimedia web page. The supplemental content maybe interactive. For example, the content may be a hyperlink on which auser can click to obtain a web page that is relevant to the mediastream. Alternatively, the content may comprise a communication link,such as a Voice over Internet Protocol (VoIP) call, or a button or otheruser interface device for placing such a call or initiating a softwaredownload.

FIG. 1 shows a flow diagram of a process for presenting context-relevantsupplemental content to a user according to one embodiment of thepresent invention. The process starts with a source system 110 (e.g., atelevision broadcasting station) transmitting a media stream 120 (e.g.,a television program) to a context detection and supplemental contentselection system (“the context system”) (represented by a dashed linebox in the FIG. 1).

The context system determines 130 context of the media stream andselects 150 content and/or advertisement for the media stream based onthe determined context. In one embodiment, the context system can alsoreceive non-media context information 140 (e.g., the user's preferences)from an external source and make the selection 150 based at least inpart on the non-media context information. The context system can add160 the selected content and/or advertisement (the context-relevantsupplemental content) to the media stream. The context system can sendthe media stream and the supplemental content to the destination system180, which can present the media stream and the context-relevantsupplemental content to the user.

System Architecture

FIG. 2 is a high-level block diagram of a system 200 according to oneembodiment of the present invention. The system 200 includes a sourcesystem 110, a context detection and supplemental content selectionsystem (“the context system”) 210, and a destination system 180. Achannel 220 exists between the source system 110 and the context system210. A channel 230 exits between the context system 210 and thedestination system 180.

The source system 110 includes hardware and/or software devices that areconfigured to transmit media content to the context system 210. It maybe a central server at a remote broadcasting location, a network accessserver, a network storage device, or a computer at a user's home.Alternatively, the source system 110 may include multiple computers asin a peer-to-peer scenario. In one embodiment, the source system 110 canbe a broadcast station (e.g., radio station and television station) thatbroadcasts media content over the air or through other channels such ascable networks. The source system 110 may provide services such as videoon demand or pay-per-view. In another embodiment, the source system 110can be an audio/video device such as a cable box, a satellite receiver,a DVD player, or a digital video recorder (DVR). In yet anotherembodiment, the source system 110 can be a time-shifting device (e.g., apersonal video recorder) or a placeshifting device (e.g., a personalbroadcaster).

In placeshifting, a user can watch or listen to live, recorded or storedmedia on a remote device via a data network. For example, a user maystream content from a first device at home (such as a satellitereceiver, cable box, or digital video recorder) to a second device (suchas a cellular phone), which can receive the media from across the house,across town, or across the world. This transmission may use public orprivate networks. A product that currently provides such a placeshiftingapplication is the SLINGBOX™ from Sling Media, Inc., and described inU.S. application Ser. No. 11/147,664, filed Jun. 7, 2005, the contentsof which are incorporated by reference in their entirety.

The context system 210 includes hardware and/or software devices thatare configured to receive media content from the source system 110through the channel 220, and determine context information of the mediacontent. In one embodiment, the context system 210 is further configuredto select context-relevant supplemental content of the received mediacontent based at least in part on the determined context information,and send (or transmit) the supplemental content and the media content tothe destination system 180 through the channel 230. In anotherembodiment, the context system 210 sends the context information to thedestination system 180, which selects the context-relevant supplementalcontent based at least in part on the received context information andpresents the supplemental content and the media content to users. In oneembodiment, the context system 210 can be configured to convert thereceived media content from one media format to another or perform anyother transcoding, encoding, translation, digitization (such asconverting media content from analog form to digital form), or otherprocessing on the media content as desired. In one embodiment, thecontext system 210 may transmit information to the source system 110.

The destination system 180 includes hardware and/or software devicesthat are configured to play the media content and the supplementalcontent to users (or viewers). The destination system 180 may includeone or more output devices, such as a television, a speaker, a personalcomputer, a personal digital assistant (PDA), a cellular phone, adedicated set top box that outputs to a television or monitor, or anyother type of device that can receive and render media data. In oneembodiment, the destinations system 180 may play the media content andthe supplemental content in the same media output device(s) or differentmedia output devices. In one embodiment, the destination system 180 cantransmit information to the context system 210 and/or the source system110.

In one embodiment, the source system 110, the context system 210, andthe destination system 180 are structured to include a processor,memory, storage, network interfaces, and applicable operating system andother functional software (e.g., network drivers and communicationprotocols).

The channel 220 is the delivery pathway through which the source system110 transmits the media content to the context system 210. The channel230 is the delivery pathway through which the context system 210transmits the media content and the supplemental content to thedestination system 180. Media content and/or supplemental content may bedelivered through a variety of pathways, including, but not limited to,the following: over the air via analog or digital radio frequency (RF)transmission; Internet protocol (IP) transmission delivered over a wirednetwork or a wireless network (or a combination) in a unicast ormulticast method; wireless or wired networks (which may or may not beIP) dedicated to the delivery of multimedia content (e.g., Qualcomm'sMediaFLO); and satellite transmission. Therefore, the channels 220 and230 may include, without limitation, one or a combination of air, wiredor wireless network, and satellite. In one embodiment, the channel 220and/or the channel 230 include the Internet. In another embodiment, thechannel 220 and/or the channel 230 may be completely within a device,such as where the context system 210 is on the same device as thedestination system 180 or the same device as the source system 110. Forexample, a computing system may be configured to stream media content tothe destination system 180 and select and send context-relevantsupplemental content to the destination system 180, thereby serving asboth the source system 110 and the context system 210.

Operation

FIG. 3 is a flowchart of a process of the context system 210 forselecting and presenting context-relevant supplemental content inaccordance with one embodiment of the invention. This process may beimplemented in software, hardware, or a combination of hardware andsoftware. In one embodiment, a part or the whole process may beimplemented in the source system 110 and/or the destination system 180.

As illustrated in FIG. 3, the context system 210 receives 310 a mediastream from the source system 110. The context system 210 thendetermines 320 context information about the media stream. Based atleast in part on the determined context information, the context system210 selects 330 supplemental content and presents 340 the supplementalcontent and the media stream via a destination system 180. Furtherdetailed for each step of the process are described below.

Receiving the Media

The context system 210 receives 310 the media stream from the sourcesystem 110. In one embodiment, the media stream is delivered to thecontext system 210 through broadcast distribution either over the air(e.g., traditional radio and television broadcasting) or through abroadcast network (e.g., cable radio and cable television). In anotherembodiment, the media stream may be delivered to the context system 210upon demand (e.g., through a video on demand service). Alternatively,the media stream may be obtained from a device that is configured toplayback locally stored media streams.

In one embodiment, the media stream is sent to the context system 210 asa series of small packets from a single source (e.g., a media server) ormultiple sources (e.g., multiple computers in a peer-to-peer networkwhere the media stream is distributedly stored). The media stream may besent using Datagram protocols. Depending on how the context system 210is configured to determine 320 the context information, the contextsystem 210 may or may not assemble the packets to construct the mediastream.

Determining the Context Information

The context system 210 determines 320 context information about thereceived media stream. Many different types of context information maybe used in embodiments of the system. For example, the contextinformation may be related to (1) the content within the media stream,(2) the destination system 180, and/or (3) the user (e.g., viewer orconsumer) of the media stream. As is described in greater detail below,the context system 210 may determine 320 the context information usingany of a variety of methods.

In one embodiment, the context system 210 has one or more desiredschemes for providing supplemental content. For example, the contextsystem 210 may be configured to provide content-relevant advertisementsfor some advertisers. These desired schemes may favor some contextinformation over others. Depending on what context information isdesired, the context system 210 may use various methods to determine 320the context information.

One type of context information relates to the content within the mediastream (content-relevant context information). This type of contextinformation may include any information about the media stream, such asthe name of a program in the stream, actors in the program, genre of theprogram. For example, for a media stream including a clip of a baseballgame, the context information may include the identity of the baseballplayer at bat.

One method to determine 320 the content-relevant context informationuses text transcripts of the media stream. Broadcasting is oftenaccompanied by text transcripts for the hearing impaired. The texttranscripts include transcript of the video program and/or advertisementthat is currently being broadcasted. These text transcripts may beincluded in metadata such as closed captioning and teletext. Dependingon the video transmission standard and/or the delivery mechanism, thetext transcripts can be transmitted using a variety of methods. Forexample, the closed captioning can be contained within the VerticalBlanking Interval (VBI) according to the NTSC standard. As anotherexample, the text transcripts can be embedded in the IP packets of themedia stream.

Because the text transcripts include a (nearly) literal transcript ofthe content in the media stream, they likely contain very usefulinformation for determining content-relevant context information. Thetext transcripts could contain information which indicates the genre ofa media program in the media stream, the title of the media program, orthe precise content or dialog that is the focus of the media programand/or advertisements at any point during the media program. Forexample, through the processing of the closed capturing transmittedduring an airing of the television program Daily Show, the contextsystem 210 can determine 320 that the viewer is watching the “DailyShow,” that the host is “Jon Stewart,” that the guest is “DavidSedaris,” that the name of the new book Mr. Sedaris is writing is “MeTalk Pretty One Day,” and that the topics covered during the show were“the war in Iraq,” “Chief Justice Roberts,” and “Wal-Mart.” In addition,the natures and sponsors of the various advertisements (e.g., “Ford” and“Mustang”) presented during the program may be similarly determined 320.

As described above, the context information may be determined 320 basedon the desired schemes for providing supplemental content. For example,if the desired schemes include advertisements for major retailers andautomobile manufacturers, the content-relevant context informationdetermined 320 for the Daily Show program will include terms such as“Wal-Mart,” “Ford,” and “Mustang,” not terms such as “the war in Iraq.”

In one embodiment, the context system 210 can determine 320 the contextinformation in real time as it receives 310 the relevant content in themedia stream. This can be important for the supplemental contentproviders. For example, a company may want to show its advertisement toa user soon after media content related to the company is presented tothe user.

Another method to determine 320 the content-relevant context informationuses text messages displayed graphically in the video content in themedia stream. In one embodiment, the context system 210 can determine320 the text messages using optical character recognition (OCR)technology. By translating the graphical text messages into recognizablewords, the text messages displayed may then be used to determine 320 thecontent-relevant context information.

Other methods of determining 320 the content-relevant contextinformation include object recognition, voice recognition, and othercontent recognition by processing the content of the media stream usinga variety of algorithms. This enables the context system 210 todetermine 320 content-relevant context information by identifyingobjects that are shown and words that are spoken in the media stream.

In one embodiment, a content provider can put a certain signal in themedia stream (e.g., an identifier in the VBI) to indicate the presenceof a specific content. The context system 210 may be configured todetect and recognize these signals and determine 320 context informationaccordingly. In another embodiment, the context system 210 may use mediafingerprints to determine 320 context information about the mediastream. A characteristic digital fingerprint may be generated for aportion of a media program (such as an audio portion of the media, or atime period thereof). The digital fingerprint may then be referencedagainst a database of fingerprints for known media items, and thereby beused to identify the media program or information about it, such asgenre, scene changes, and specific programming ID of the media program.

In one embodiment, a content provider can transmit context information(e.g., media fingerprint, text transcripts) in the audio signals of themedia stream. For example, the content provider can transmit a contentindicator in the subaudible tone (or other audio tone that is notnoticeable to the average listener at normal volume levels) of an audiostream. The context system 210 may be configured to detect and recognizethe context information embedded in the audio signals, and thereby useit to determine context information about the received media stream.

The context system 210 may use information from sources other than themedia stream to determine 320 the content-related context information.Such sources may be third-party databases, such as web services andelectronic program guides (EPG). An EPG contains descriptive informationabout the content of programs in multiple channels. The context system210 may retrieve descriptive information from the EPG usingidentification information such as the date and time when the program isbroadcasted, the location of the destination system 180, the serviceprovider and subscribed package (e.g., Comcast basic cable), and thecurrent channel being viewed. The context system 210 may retrieve theidentification information from the destination system 180 and/or thesource system 110. In one embodiment, the source system 110 is aplaceshifting device which can provide such information to the contextsystem 210 along with the media stream.

Another type of context information that may be used to select 230supplemental content related to the user, the destination system 180, orother information that is not determined solely by the media stream.This type of context information may be called usage-relevant contextinformation (or non-media context information). Examples ofusage-relevant context information include configurations andcapabilities of the destination system 180 (e.g., type and model ofequipped output device, operating system), time and location of theoutput device playing the media stream, and information about the user(e.g., web browsing histories of the user). The context system 210 mayretrieve the usage-relevant context information from the source system110, the destination system 180, or some other source. For example, thecontext system 210 may determine 320 the location of the destinationsystem 180 using the IP address of the destination system 180. Asanother example, the context system 210 may retrieve user registrationdata from the source system 110. Assuming the destination system 180includes a personal computer, the context system 210 may use softwareprograms or data located on the computer (such as web histories andInternet cookies) to determine 320 context information about the user.

The context system 210 may also derive additional contextual informationbased on the determined context information. For example, knowing thatthe viewer lives in one state (e.g., through user registration data) andis presently viewing media content in another state (e.g., through IPaddress of the destination system 180), the context system 210 may inferthat the viewer is traveling. The inferred context information may beused to select 330 supplemental content that would be highly relevantfor a traveler, such as a local restaurant guide for travelers.

Therefore, the context system 210 may determine 320 the content-relevantcontext information (derived from the media stream itself) and/or theusage-relevant context information (derived from sources other than themedia stream) of the media stream.

Selecting Supplemental Content

Once the context information is determined 320, the context system 210selects 330 supplemental content using that context information. In oneembodiment, the context system 210 is configured to select 330 thesupplemental content that is most relevant to the user, and thus mostvaluable to parties of interest (e.g., the user, supplemental contentproviders).

In one embodiment, the context system 210 uses the context informationto select 330 the supplemental content from a collection of supplementalcontent. The collection of supplemental content may includeadvertisements or other information that may be relevant to the media.The collection of supplemental content may be stored locally in thecontext system 210 or externally in devices such as third-partydatabases and the destination system 180.

In one embodiment, the context system 210 conducts the selection 330based at least in part on the content-relevant context information.Supplemental content providers (e.g., advertisers) may prefer to providetheir supplemental content (e.g., advertisements) with media programsthat are relevant in some way to the thing being advertised. Forexample, a company may want to present its own advertisement to theviewer whenever its competitor's advertisement appears. As anotherexample, a company may want to supplement its own advertisements withadditional information or timely offers. Beneficially, the users aremore likely to be interested in supplemental information that isrelevant to the content of the media program they are watching. Forexample, a viewer would be more likely to be interested in a briefbiography of Chief Justice Roberts if the viewer is watching atelevision program about the judicial nomination of John Roberts.

In one embodiment, the context system 210 conducts the selection 330based at least in part on the usage-relevant context information. In oneembodiment, the output device of the destination system 180 presenting340 the media stream may influence the selection 330 of the supplementalcontent. For example, the context system 210 may select 330 a cellularphone plan advertisement if the user is watching the media via acellular phone, but select 330 a cable service advertisement if the useris watching via a television or personal computer. In anotherembodiment, the inventory of the destination system 180 may impact thesupplemental content the context system 210 selects 330. For example, ifa computer of the destination system 180 has a particular operatingsystem installed, the context system 210 may select 330 an advertisementof a compatible software application which may otherwise not be selected330. In another embodiment, the past activities of the user may drivethe selection 330. For example, cookies on a user's computer mayindicate the user's preferences, which may be considered by the contextsystem 210 in selecting 330 the supplemental content to be presented 340to the user.

In one embodiment, the context system 210 selects 330 the supplementalcontent based at least in part on statistical data of other users'activity. Supplemental content successful in similar context in the past(e.g., an advertisement with high click rate) tends to remain successfulin the near future. Therefore, the context system 210 may select 330 thesupplemental content with favorable statistical data over theunfavorable ones, thus increasing the effectiveness of the selectedsupplemental content. In one embodiment, the context system 210 can usealgorithms such as the Bayesian inference to determine a success scorefor candidate supplemental content items, and select 330 the one withthe highest score.

In one embodiment, the context system 210 implements the function forselecting 330 supplemental content as a rule engine. The rule engine mayinclude rules that correspond to logics (or algorithms) the contextsystem 210 uses to conduct the selection 330. The rules may be providedby parties of interest. A rule may be based on content-related contextinformation. For example, a rule may be to select 330 an airlineadvertisement whenever a program contains the word “Hawaii” two or moretimes in a media stream. The matching (or filtering) of the words may beliteral (e.g., “Cadillac” should trigger content associated with“Cadillac”) or associative (e.g., the word “basketball” may beassociated with content related to sneakers and other shoes). As anotherexample, a user may set a rule that prohibits the selection 330 of anyadvertisement for a media program about jewelry. A rule may be based onusage-related context information. For example, an advertiser may set arule that prohibits the selection 330 of its advertisements when thedestination system 180 uses a cellular phone to present 340 thesupplemental content. The rules engine thus allows for any rule orcombination of rules for the context system 210 to select 330supplemental content from the collection of context information.

In one embodiment, after selecting 330 the supplemental content, thecontext system 210 selects a format for the selected supplementalcontent. The supplemental content may have multiple formats, each ofwhich may be presented differently based on the destination system 180.For example, an advertisement for a particular product may have a videoformat having a high-resolution video version and a low-resolutionversion, an audio format for radio listeners, and a web format having anURL pointing to a web page designated to promote the product.

The context system 210 may select a format for the supplemental contentbased on the context information. A number of factors may be consideredin selecting the format of the supplemental content, such as thelocation of the user, the configuration of the destination system 180,and the format of the media stream. For example, if the user placeshiftsvideo stream from home and views the stream using a laptop computer, thecontext system 210 may select a shortened low-resolution format of theselected supplemental content. However, if the viewer is watching thesame video stream at home using a large screen high-definitiontelevision (HDTV), the context system 210 will select a format that candeliver full-length high-resolution of the selected supplemental contenton the HDTV. If the context system 210 discovers that the user iswatching the same video stream using a mobile wireless device (e.g., asmart phone), it may select an abbreviated format that will be displayedas a scrollbar in the device.

In one embodiment, the context system 210 implements the function forselecting a format for the supplement content using the same rule engineas the one for the function for selecting 330 the supplemental content.Parties of interests (e.g., users or advertisers) may add or modifyrules to set their preferences. For example, some advertisers may prefertheir advertisements to be in a particular format (e.g., text only),while other advertisers may prefer to display their advertisementtogether with the media stream and launch a particular web page for theviewers to visit.

In one embodiment, the selection 330 of the supplemental content isconducted by the destination system 180. The context system 210 may send(or transmit) the determined context information to the destinationsystem 180. The destination system 180 may make the selection 330 inmanners similar to the ones described above.

Presenting the Supplemental Content

Once the supplemental content is selected 330, the supplemental contentand the media stream are presented 340 to users via the destinationsystem 180. In one embodiment, the context system 210 sends (ortransmits) the media stream and the supplemental content to thedestination system 180 using the same communication channel 230. Thesupplemental content may accompany the media stream synchronously orasynchronously.

Depending on the formats of the media stream and the supplementalcontent, the context system 210 may incorporate the supplemental contentinto the media stream to form a new combined media stream and send thecombined stream to the destination system 180. For example, if both themedia stream and the supplemental content include video content, thecontext system 210 may add video content of the supplemental content infront of the video content of the media stream (pre-placement), at theend of the video content of the media stream (post-placement), or abovethe video content of the media stream (overlay). Alternatively, thesupplemental content may be delivered in a different channel and/or bycompletely different delivery means than the media stream. For example,while the media stream may be sent to the destination system 180 througha computer network, the context system 210 may send 340 the supplementalcontent offline. For example, the supplemental content may be sent 340through a phone call, a fax, or the postal service to the user's home.The destination system 180 may combine (or incorporate) the supplementalcontent and the media stream to present 340 them to the users.

In one embodiment, the supplemental content is stored in the destinationsystem 180 or in a storage device accessible by the destination system180. Rather than sending the supplemental content to the destinationsystem 180, the context system 210 may send information necessary forthe destination system 180 to access the supplemental information. Forexample, the destination system 180 may have certain advertisementsstored in its local storage. Rather than sending an advertisement to thedestination system 180, the context system 210 may send an identifierthat uniquely identifies the advertisement in the local storage. Asanother example, the context system 210 may send a uniform resourcelocator (URL) associated with the supplemental content to thedestination system 180, which may then retrieve the supplemental contentusing the URL through a network. Beneficially, this approach alleviatesmuch of the need for transmission of supplemental content from thecontext system 210 to the destination system 180.

In one embodiment, the context system 210 only sends the supplementalcontent to the destination system 180 and not the media stream. Inanother embodiment, the context system 210 stores (or caches) the mediastream and the supplemental content in a local storage device and sendsthe media stream along with the supplemental content to the destinationsystem 180 at a later time. For example, the context system 210 may alsofunction as a time-shifting or placeshifting device. The context system210 may include personal video recorder functionality (using localstorage to record the media stream received) and tag the recorded mediastream with information indicating the content of the programs recorded.On play back, this information can be used to select supplementalcontent such as context-relevant advertisements and information.

Depending on the format of the supplemental content, the format of themedia stream, and/or the configuration of the destination system 180,the destination system 180 may present 340 the supplemental content tothe users in a variety of ways. The destination system 180 may present340 the supplemental content using the same output device as the oneused to present 340 the media stream. For example, the destinationsystem 180 may display both the supplemental content and the mediastream in the same screen. The supplemental content and the media streammay also be presented 340 using the same application. For example, thedestination system 180 may display a television program (media stream)using a media player, and overlay an advertisement (supplementalcontent) as a scrollbar (or text ticker) on the television program usingthe media player. Alternatively, the supplemental content and the mediastream may be presented 340 using different applications. For example,if the supplemental content includes a URL, the destination system 180may launch a web browser to display the web page associated with theURL, while displaying the media stream using a media player. One exampleof the media player is described in more detail in U.S. patentapplication Ser. No. 11/147,664, filed on Jun. 7, 2005, the content ofwhich are hereby incorporated by reference.

The destination system 180 may present 340 the supplemental content andthe media stream using different output devices. For example, thedestination system 180 may present 340 a television program (mediastream) on a HDTV and print a coupon (supplemental content) using anassociated printer.

In one embodiment, the destination system 180 and/or the context system210 replaces a portion of the media stream with the supplementalcontent. For example, the destination system 180 may replace anadvertisement for a company with a different advertisement for thecompany that is determined to be more relevant to the user.

Example Process

The process described above can be further illustrated through anexample of an operation of the context system 210 in accordance with oneembodiment of the invention. In this example, a user is watching atelevision broadcast of a baseball game using a personal computerthrough a placeshifting device.

While receiving the television broadcast from a television station, theplace shifting device determines that a particular baseball player is atbat. Based on this information, the placeshifting device selects anadvertisement for a website that sells sports memorabilia. To do this,the placeshifting device may select a URL for a web page on the websitewhere the viewer can buy an autographed ball from the baseball playerthat is currently at bat. This selection may also be based in part oninformation that is not determined from the media stream, such as theuser's previously stated preferences or activity (e.g., the user hasvisited websites that have this player's stats, thus indicating aninterest in the player). This supplemental content is then delivered tothe user's personal computer with the television broadcasting. As aresult, while the user is watching his favorite player at bat during abaseball game on the personal computer, the user also receives anadvertisement for the relevant sports memorabilia as well as a link to aweb page where the user can easily buy the advertised product.

Advertisement Marketplace Business Model

The systems and methods described above enable a business model in whichthird parties can register advertising or supplemental content andspecify the criteria that trigger it. Context, as defined above, mayplay a large part in those criteria. The third parties' criteria areused, at least in part, to select the supplemental content to bepresented with the media stream. When a third party's advertisement isplayed, presented or interacted with, the third party is charged a fee.The amount charged may be based on the number of times the advertisementis selected and added to a media stream, for each viewer. In oneembodiment, the service includes a web interface for third parties toupload their advertisements or other supplemental content and select ordefine the criteria to be applied for pairing the supplemental contentwith the media content presented to viewers.

The service may also allow the third parties to bid amounts of money oncertain criteria, where the bid amount is or is related to the pricethat the third party is charged for the presenting of its content withthe media stream. The system may then give priority to third parties'content based on the associated bid amount. This enables an efficientmarketplace for advertisements, where prices adapt to a market price forvarious criteria and content based on demand.

In another aspect of an embodiment of this model, the service that pairsthe supplemental content with the original television or video contentcan be independent from all the primary and secondary content providers.The technologies used to determining context while transporting primarymedia may serve as a matchmaking virtual marketplace for third partiesinterested in delivering their supplemental information to a particularset of users. In addition, the transport providers (e.g., the carriersand others) might be included in this model in a variety of ways basedon the type of content presented to the user.

In one example of this, an advertising agency may wish to place onemillion dollars of advertising on computers and mobile devices for itsclient. By bidding on keywords (skateboard, motocross, etc), genres (forexample: sports::extreme) and indicating that it wants to place videocontent on computers and mobile devices, a spend plan is built aroundthe budget using the service described above. Once buys are in place andconfirmed, whenever a 19 year-old placeshifts the X-Games onto hiscellular phone, the client is assured that a certain number of airingswill include their video advertisement. Money paid by the client may bedistributed in different percentages among the primary content owners,the service that enables this business, and the cellular phone carriers,all based upon the tracking data maintained by the enabling service.

The tracking information that can be maintained by the context/contentmatching business includes performance metrics, including, but notlimited to: (1) how often secondary content was presented to users; (2)how often they interacted with the secondary content; (3) where and whenthe secondary content was presented; (4) which primary content triggeredthe secondary content; and (5) which networks carried the secondary andprimary content. Based on these metrics, the different parties involvedcan agree upon appropriate ways to split up the revenue stream.

Summary

Embodiments of the invention provide a system and method to pair mediacontent with context-relevant supplemental content and present the mediacontent and the supplemental content to a user.

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the aboveteachings.

Some portions of above description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

In addition, the terms used to describe various quantities, data values,and computations are understood to be associated with the appropriatephysical quantities and are merely convenient labels applied to thesequantities. Unless specifically stated otherwise as apparent from thefollowing discussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” or“calculating” or “determining” or the like, refer to the action andprocesses of a computer system or similar electronic computing device,which manipulates and transforms data represented as physical(electronic) quantities within the computer system memories or registersor other such information storage, transmission, or display devices.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a computer readable storage medium, such as, but notlimited to, any type of disk including floppy disks, optical disks,CD-ROMs, magnetic-optical disks, read-only memories (ROMs), randomaccess memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards,application specific integrated circuits (ASICs), or any type of mediasuitable for storing electronic instructions, and each coupled to acomputer system bus. Furthermore, the computers referred to in thespecification may include a single processor or may be architecturesemploying multiple processor designs for increased computing capability.

Embodiments of the invention may also relate to a computer data signalembodied in a carrier wave, where the computer data signal includes anyembodiment of a computer program product or other data combinationdescribed herein. The computer data signal is a product that ispresented in a tangible medium and modulated or otherwise encoded in acarrier wave transmitted according to any suitable transmission method.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems may also be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description above.In addition, embodiments of the invention are not described withreference to any particular programming language. It is appreciated thata variety of programming languages may be used to implement variousembodiments of the invention as described herein, and any references tospecific languages are provided for disclosure of enablement and bestmode of embodiments of the invention.

Finally, it should be noted that the language used in the specificationhas been principally selected for readability and instructionalpurposes, and may not have been selected to delineate or circumscribethe inventive subject matter. Accordingly, the disclosure of the presentinvention is intended to be illustrative, but not limiting, of the scopeof the invention, which is set forth in the following claims.

What is claimed is:
 1. A method executable by a data processing system,the method comprising: receiving a clip of media content from a firstmedia player device at the data processing system, wherein the clip ofmedia content is a clip extracted from a broadcast television program bya user of the first media player device; analyzing at least a portion ofthe received clip of media content by the data processing system tothereby automatically identify the broadcast television program fromwhich the user-created clip originated, wherein the analyzed portion ofthe received clip comprises metadata that identifies the broadcasttelevision program that is obtained from an electronic program guide(EPG) by the first media player device, and wherein the first mediaplayer device embeds the metadata obtained from the electronic programguide in the clip of media content for subsequent analysis by the dataprocessing system; identifying supplemental content related to theidentified clip of media content; and providing the identified clip ofmedia content and the supplemental content over the network to a secondmedia player device for presentation to a second user.
 2. The method ofclaim 1 wherein the analyzing comprises analyzing the media content bythe data processing system in real time as the dip of media content isreceived at the data processing system.
 3. The method of claim 1,wherein the supplemental content comprises an advertisement.
 4. A methodexecutable by a data processing system that processes media content, themethod comprising: receiving a clip of media content from aremotely-located media client device via a network, wherein the receivedclip comprises a portion of media content extracted from a broadcasttelevision program by a user of the remotely-located media clientdevice; analyzing at least a portion of the received clip of mediacontent at the data processing system to automatically determinecontext-relevant information that identifies the broadcast televisionprogram from which the user extracted the clip, wherein the analyzedportion of the received clip comprises metadata that identifies thebroadcast television program and that is obtained from an electronicprogram guide (EPG) by the remotely-located media client device, andwherein the remotely-located media client device embeds the metadataobtained from the electronic program guide in the clip of media contentfor subsequent analysis by the data processing system; processing thecontext information to thereby select supplemental content that isrelated to the identified content using the data processing system; andproviding the supplemental content and the identified media content forpresentation to a viewer of a second remotely-located client device viathe network.
 5. The method of claim 4 wherein the supplemental contentis an advertisement.
 6. The method of claim 4 wherein the supplementalcontent is an advertisement for a product, and wherein the product isrelated to the content of the clip extracted by the user.
 7. The methodof claim 4 wherein the supplemental content comprises an advertisementselected from a plurality of advertisements based upon the determinedcontext information so that the selected advertisement advertises aproduct that is related to the clip of media content.
 8. A dataprocessing system comprising: an interface to a network; and a processorconfigured to receive a clip of media content from a remotely-locatedclient device via the network, wherein the clip is extracted by a userof the remotely-client device from a broadcast television program, toanalyze the received clip to thereby automatically identify thebroadcast television program from which the clip was extracted, and toidentify supplemental content related to the clip, and to provide theidentified clip and the supplemental content to a secondremotely-located client device via the network for presentation to aviewer, wherein the processor is further configured to analyze metadatacontained in the received clip of media content that identifies thebroadcast television program, and wherein the metadata is obtained froman electronic program guide (EPG) by the remotely-located client device,and wherein the remotely-located client device embeds the metadataobtained from the electronic program guide in the clip of media contentfor subsequent analysis by the data processing system.
 9. The dataprocessing system of claim 8 wherein the remotely-located client deviceobtains the broadcast television program as a placeshifted media streamfrom a placeshifting device, and wherein the remotely-located clientdevice extracts the clip from the placeshifted media stream based uponinputs received from the user of the remotely-located client device thatdefine a beginning and an end of the placeshifted media stream.